Both video and motion-picture content, and non-moving visual content like a printed magazine with colored still images, is currently sold to end consumers in one of two distinct visual formats. The dominant and older format delivers visual content to consumers as flat 2D images and video. The newer but less common format delivers images and video content to the end consumer in stereoscopic 3D. At the time this patent was written (Q2 2014), there was far more 2D content available on the global content market than 3D content. Out of thousands of video- or motion-picture titles available to purchase on High Definition Bluray discs, for example, only around 280 titles—a fairly small percentage of the total—were stereoscopic 3D titles sold as “Bluray 3D” content. The vast majority of content available to buy on DVDs and Bluray discs is presently in flat 2D. Also, virtually all video and filmed content produced before the year 2003—the year the “New 3D Cinema” paradigm first started to be experimented with—is available only in 2D form. While it is currently possible to purchase a fairly sophisticated Stereoscopic 3D capable TV set with accompanying 3D glasses for less than 1,000 Dollars, the so-called “Content Problem”—the limited availability of engaging content mastered or produced in Stereoscopic 3D—has caused the “3D Cinema At Home” market strategy of large consumer electronics manufacturers to fail in financial terms. While Stereoscopic 3D films screened at cinemas have enjoyed some popularity and financial success—2009's flagship 3D film “Avatar” used stereoscopic 3D so expertly that it quickly became the highest grossing film in history—the limited choice of engaging stereoscopic 3D content available for viewing at home has resulted in consumers buying far fewer 3D capable TVs, since their introduction, than the industry hoped and predicted would be the case. The lower-than-expected sales numbers of 3D TVs have been so disappointing to TV manufacturers that the industry, by and large, no longer promotes the 3D-viewing aspect of LCD and LED TVs actively, and also no longer charges a premium price for 3D capable TVs, as was the case when 3DTVs were first introduced. At the time of writing, the TV manufacturing industry has virtually stopped promoting 3D Television sets, and hopes, instead, that UHD (“Ultra High Definition”) capable TV sets with a never-before-seen screen resolution of 4K or 8K, will sway consumers to upgrade to a new TV, instead of 3D viewing functionality being used to attract potential TV buyers. The new UHD TV sets do come with stereoscopic 3D viewing capability built-in. But the industry no longer emphasizes the Stereoscopic 3D capability in selling these next-generation TVs, arguing instead that moving Ultra High-Definition pictures at 4K or 8K resolution make for such realistic and 3 dimensional viewing in plain 2D, that stereoscopic 3D and the associated 3D glasses are no longer required to facilitate an immersive, 3 dimensional and life-like viewing experience.
One way to solve—or perhaps more accurately“alleviate”—the “3D Content Problem” is to convert visual content shot in traditional 2D to Stereoscopic 3D in a “post process conversion”. This process is commonly called “2D-to-3D Conversion”, “3D Conversion” or—in the case of some Hollywood movies—“Dimensionalization”. 3D Conversion of 2D content can be achieved in two ways. “Manual” or “artist-driven” 3D conversion, an expensive process used mostly for high-budget content like major Hollywood movies, relies on trained human operators to tell a 3D conversion software, typically on a frame-by-frame or shot-by-shot basis, what portion of a 2D frame is at what distance and angle relative to the virtual stereoscopic 3D camera created by the 3D conversion software. Manual 3D conversion is currently a slow, labour-intensive and expensive process. The popular 1990s 2D film “Titanic”, for example, was manually converted to Stereoscopic 3D over 14 months, by 450 trained 3D conversion artists working on frame-by-frame or shot-by-shot basis, with the final 3D converted version of the film costing 18 Million Dollars to complete. At the time of writing, a fully“manual” or “semi-automatic” artist-driven 3D conversion can cost anywhere from 1,500 Dollars to 100,000 Dollars per minute of footage converted, depending on the company performing the conversion, the visual quality of the stereoscopic 3D end product achieved, and the specific 3D conversion method, process, software or technology used. The quality of 3D conversion achieved by an artist-driven process can be very good in technical and artistic terms. But the slow turnaround times and high cost of the process—several million dollars and many months of time expended to convert just one 2 hour feature film to 3D—makes it feasible to use only on select films that are popular enough for the 3D version to generate more revenue, from cinema ticket sales and Bluray 3D discs sold, than the manual 3D conversion process cost to complete. This effectively means that many 2D films that do not fall into the category“highly popular”, will never get converted to 3D, or at least not until the manual 3D conversion process becomes faster and cheaper.
The second way to convert 2D content to 3D is via the use of a fully automated 2D-to-3D conversion algorithm, or a realtime 2D-to-3D conversion hardware chip. Here, no human operator is involved. Instead, an automatic 2D-to-3D algorithm or 3D conversion method applies various mathematical operations to a flat 2D video frame, in order to generate a Stereoscopic 3D version of that 2D frame. Many 3DTVs and some Bluray players and living-room media players come with some form of realtime 2D-to-3D conversion technology built in. This technology is usually implemented in the form of a fast hardware chip that converts from 2D to 3D “on the fly”—2D content is converted to stereoscopic 3D in realtime, as one is viewing the content. The quality of 3D visuals achieved by automated 2D-to-3D hardware varies by manufacturer, device, and the sophistication of the technical approach used. The current generation of automated 2D-to-3D chips and algorithms is capable of creating “watchable” 3D content from some 2D content—a film, a football match, for example, or perhaps downhill skiing, or Formula One racing. But the general consensus amongst owners of current generation 2D-to-3D conversion capable 3D TVs and Bluray and media players is that automated 2D-to-3D does not, at present, yield the kind of “sophisticated Stereo 3D effect” seen in high-budget content that was either shot in genuine stereo 3D, or shot in 2D and put through an expensive, manual, artist-driven 2D-to-3D process under the supervision of a trained Stereographer. So while 2D-to-3D conversion hardware and software is capable of creating a 3D effect from 2D footage, the 3D effect created is typically not as high quality, visually and artistically speaking, as the more sophisticated 3D effect created by either filming in genuine stereo 3D, or by post-converting via artist-driven, manual or semi-automatic 2D-to-3D conversion.
To summarize: When 2D content is to be converted to 3D, manual 2D-to-3D conversion currently yields the highest quality stereoscopic 3D, but is so labour-intensive and expensive at present that it can only be applied to a handful of films popular enough to offset the high conversion cost when screened or sold to the end user. Automatic 2D-to-3D conversion, on the other hand, is quick, inexpensive and practical, and allows any content, no matter how old or new, popular or obscure, to be viewed in stereoscopic 3D, instantly. But the quality of 3D effect generated this way cannot, at present, rival the results of an expensive, shot-by-shot, human-supervised manual 3D conversion. The owners of some manual 3D conversion companies have gone on record saying “A living, breathing, thinking human artist will always produce better Stereoscopic 3D from 2D source material than an automated algorithm or 3D conversion hardware chip can hope to achieve”. While this may be somewhat true at present, there are many real-world applications where the only thing that makes sense is cheaper and faster realtime—or offline—automated 2D-to-3D conversion. Hence, R&D and development of automated 2D-to-3D conversion techniques continues at a brisk pace, with automated 3D conversion techniques becoming a little more sophisticated with each year that passes.
The 2D-to-3D conversion method presented in this patent falls into the category of automated, unsupervised, realtime capable 2D-to-3D conversion methods. It is intended to be deployed in the real world as either a dedicated realtime 2D-to-3D hardware chip, or as a realtime 2D-to-3D software algorithm running on fast graphics processing hardware like a Graphics Processing Unit (a “GPU”).